Your browser doesn't support the features required by impress.js, so you are presented with a simplified version of this presentation.

For the best experience please use the latest Chrome, Safari or Firefox browser.

Screenshot of the article

Context

Kyle Erf, 2016:
Evergreen Continuous Integration: Why We Reinvented The Wheel
Ingo & Daly, 2020:
Automated System Performance Testing at MongoDB
Ingo & Daly, 2019:
Reducing variability in performance tests on EC2: Setup and Key Results
David Daly, et.al. 2020:
Change Point Detection in Software Performance Testing

Requirements

  1. Full end-to-end automation
  2. Support both CI and manual benchmarking
  3. Elastic, public cloud infrastructure
  4. Everything configurable
  5. All configuration is via a single, consistent, YAML based system
  6. Clear modularity for executables and configuration
  7. Diagnosability
  8. Repeatability
Architecture Diagram
Listing 1: mongodb_setup.yml
Listing 2: test_control.yml
Evergreen CI result page

Some interesting choices

Results